Neural network approximation: Three hidden layers are enough

A three-hidden-layer neural network with super approximation power is introduced. This built the floor function (⌊x⌋), exponential (2x), step (1x≥0), or their compositions as activation in each neuron and hence we call such networks Floor-Exponential-Step (FLES) networks. For any width hyper-parameter N∈N+, it shown that FLES max{d,N} three hidden layers can uniformly approximate a Hölder continuous f on [0,1]d an rate 3λ(2d)α2−αN, where α∈(0,1] λ>0 are order constant, respectively. More generally for arbitrary modulus of continuity ωf(⋅), constructive 2ωf(2d)2−N+ωf(2d2−N). Moreover, extend result to general bounded functions set E⊆Rd. As consequence, this new class overcomes curse dimensionality when variation ωf(r) r→0 moderate (e.g., ωf(r)≲rα functions), since major term be concerned our essentially d times N independent within continuity. Finally, analysis derive similar results Lp-norm p∈[1,∞) via replacing by functions.